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Abstract —We present a method for simultaneous Receding 
Horizon Estimation and Control of a robotic arm equipped 
with a sword in an adversarial situation. Using a single camera 
mounted on the arm, we solve the problem of blocking a 
opponent’s sword with the robot’s sword. Our algorithm uses 
model-based sensing to estimate the opponent’s intentions from 
the camera’s observations, while it simultaneously applies a 
control action to both block the opponent’s sword and improve 
future camera observations. 

I. Introduction 

Today’s powerful sensors, such as integrated inertial- 
measurement units and high-speed cameras, together with 
efficient embedded processors, allow robots to perceive the 
environment with a level of detail that is at least comparable 
to human perception capabilities. But even using precise and 
fast sensors, robots are still several steps behind the capa¬ 
bilities of humans at solving everyday tasks. Indeed, even 
though prototype robots performing tasks such as driving a 
car [1], [2] or doing chores [3] exist today, their performance 
is still not comparable to that of a human when provided 
with the same perceptual stimulus. Among the reasons why 
robots cannot perform similarly to humans when confronted 
with similar perceptual stimuli, two arguments stand above 
the rest; first, humans perform active perception [4], i.e., 
we actively seek to improve the quality of the data received 
by our sensors, for example, when we move our head to 
observe a wider field-of-view; second, humans can extract 
the actionable information [5], [6] from continuous streams 
of data in a natural way, i.e., a loud noise (continuous sound 
signal) is quickly transformed into a clue of possibility of 
danger (discrete symbol) by our brain [7]. 

In this paper we design a strategy to allow a robotic arm 
equipped with a sword to block its opponents attacks [8]. 
Using a single camera mounted on one of the links of the 
robotic arm as the only perceptive information, we provide 
a practical answer to the problems of performing active 
perception and extracting the actionable information from 
the captured data. We chose to use a single camera as a 
sensor since it is passive, small, easy to calibrate, and it is 
theoretically possible to reconstruct 3-D information using 
the motion of the arm [9]. 

We implement two methods, one based on Receding 
Horizon Control (RHC) to move the robotic arm as a function 
of the position of the opponent’s sword, and one based 


The authors are with the Department of Electrical & Systems Engineer¬ 
ing, Washington University in St. Louis, St. Louis, MO 63130. Emails: 

ignacio. de . erausquin@gmail. com, h.gonzale@wust 1. edu 


on Receding Horizon Estimation (RHE) to obtain the 3- 
D coordinates of the opponent’s sword given the images 
captured by the camera. More importantly, the RHC method 
performs active perception by adjusting the position of the 
arm to improve the point of view of the camera, and our 
RHE method extracts the actionable information by using a 
dynamical model of the opponent’s movements. 

Several groups have studied the interactions between 
robots and humans, and have proposed ways to extract 
information from motion signals [10], [11], [12], [13]. In the 
context of their papers, our results should be understood as 
a particular approach towards the information retrieval using 
dynamical models and optimization-based algorithms. 

Several groups have used Receding Horizon Control & 
Estimation to process data from sensors and solve problems 
in robotics. RHE methods have been used extensively in the 
literature as state observers [14], [15], [16]. Our approach, as 
described in Section III is not to observe all the opponent’s 
state variables, but rather to estimate the opponent’s speed 
and direction of attack. Even though the problems are similar, 
our problem involves a smaller number of variables, greatly 
simplifying the numerical computations. Prazenica et al. [17] 
use several cameras to estimate the position of moving 
robots controlled by RHC. Instead of taking that approach, 
which would imply that the robotic arm is equipped with a 
collection of cameras all around the opponent, we preferred 
the approach where a single camera is actively moved, 
allowing for a compact, more realistic scenario. 

Chipalkatty et al. [18], [19] consider the problem of incor¬ 
porating human inputs into the control loop. Even though in 
our problem the opponent is assumed to be a human subject, 
we focus on the design of a completely autonomous fencing 
robot. 

Kunz et al. [20] design a full hybrid dynamical model 
describing the fencing game, where discrete modes represent 
the different intentions by the opponents, and the continuous 
variables describe the configuration space of the two arms. 
Their result is complementary to our paper. While they 
focus on a complete description of the fencing game without 
considering how the opponents observe each other, we focus 
on the interplay between perception and actions, but using 
only a defensive strategy for the robotic arm. 

Liang et al. [21] also designed a robot for fencing training. 
However, their result focuses on the components of the 
problem that do not involve moving the robot’s sword, but 
rather emphasize training the fencer’s footwork and speed. 
Our paper deals entirely with the problem of controlling the 
sword, thus we do not consider the lateral movement of the 
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two fencers. 

It is also worth noting that we currently do not claim 
any theoretical results regarding the observability of the 
opponent’s sword using a single camera, as studied by 
Hernandez and Soatto [22]. Instead, we try to focus on the 
proof-of-concept where information from a camera can be 
naturally transformed into useful (or actionable) information 
using RHE. 

Our paper is organized as follows. Section |n] presents the 
problem of designing a fencing robot in detail, as well as 
our simplifying assumptions. Section shows the imple¬ 
mentation of the RHE method which receives the camera- 
frame position of the opponent’s sword and outputs the speed 
and direction of the opponent’s attack. Section IV shows the 
implementation of the RHC method which moves the arm 
to both defend and improve the point of view of the camera. 
Section [V| shows simulations of the closed-loop system and 
analyzes its performance. Einally, Section VI presents our 
conclusions and future work directions. 


H. Problem Description 

In this section, we describe the model for sword fighting, 
or fencing, between two players, called fencers. We further 
describe the mathematical model of our robotic arm and 
camera and how these are used in the fencing problem. 

Our model for fencing is based on a simplification of 
the Olympic saber fencing rules [8]. One fencer, the at¬ 
tacker, attempts to perform an attack, while the other fencer, 
the defender, attempts to interrupt this attack. An attack 
is described as an uninterrupted motion of the attacker’s 
sword towards the target area. An attack is considered to be 
interrupted if the defender forces contact between the two 
fencers’ swords, or if the attacker stops his forward motion 
towards the target. In this paper, we will only consider the 
problem from the defender’s perspective. Therefore, the robot 
will always be the defender, while the opponent will be the 
attacker. 


A. Robot Model 

The fencing robot is an arm consisting of 6 ro¬ 
tary joints, with corresponding coordinate frames denoted 
{xd,i,yd,i,Zd,i) G for each i e {1,...,6}, as shown 
in Eigure [T] The subscript d indicates that these frames 
correspond to the defender. To simplify our notation we will 
denote the sixth-joint coordinate frame simply as {xd, yd, Zd). 
The coordinate frames are set such that the z-th joint rotates 
around Zd,i with angle 9d,i G K. Also, let (io, yo, io) bo the 
world frame, which is assumed static. All points in the paper 
will be represented in the world coordinate frame unless 
otherwise noted. 

We describe the displacement of points in 3-D space using 
the space of special Euclidean 3-D transformations, SE{3), 
defined by: 

SE{3) = {g = {R,T) G S'0(3) X K^}, (1) 

where T € represents a translation and R G SO(3) 
represents a rotation, typically described using a rotation 



Fig. 1. Diagram of the robotic arm, the coordinate frames defined by each 
of the joints, the global coordinate frame, the location of the camera, and 
the location of the sword. Each joint rotates around the i-axis (in color 
blue). 


matrix which leads to the following definition: 

50(3) = {i? G I i? = J, det(i?) = -fl}. (2) 


In the case of our robot, we define 6 transformations, denoted 
{gd,i}^^i C SE{3), each transforming points from the 
(z — l)-th frame to the z-th frame, where we abuse notation 
and set 0-th frame as the world frame. Note that, since 
the robot’s segments have a fixed length, each of these 
transformations is parameterized by the joint’s angle, i.e., 
gd,i{0d,i) for each z e {1,..., 6}. 

Throughout the paper we use the homogeneous repre¬ 
sentation for points in space and elements in SE(3), as 
described in Chapter 2 of [9]. Given a point p G and 
a transformation g = {R,T) G SE(3), then when we apply 
g to p we get a new point g = Rp -\-T. The homogeneous 
representation summarizes the algebraic operation above by 
embedding p into as [^] and embedding g into as 

[o Tl’ thus: 
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We will abuse notation and assume that all points in 
and transformations in SE{3) are in fact written in their 
homogeneous representation, e.g., we will write equation ([^ 
simply as q = gp. 

The defender’s sword extends along the z^-axis, as shown 
in Eigure which in world coordinates is given by: 


U&d) = GdiQd) [o], 


(4) 


where: 
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GdiQd) = W_9d,i{Sd,i), 

i=l 


(5) 


and Qd = [dd,i, ■ ■ ■ ■ Similarly, the base and tip of 

the defender’s sword are described in world coordinates by 















Gd{Qd)Pb and Gd{Od)pt, respectively, where: 
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are represented in sixth-joint coordinates, and (. is the length 
of the sword. 

We model the robot’s target area, i.e., the area being 
defended from the attacker, as a finite set of points of interest 
denoted ^target- 

We use a kinematic model for the robotic arm, and we 
assume that the arm is fully actuated. Hence, the joint angles 
satisfy the following ODE: Qd{t) = Ud(t), where Ud(t) € 
are the angular velocities of the joints. 


B. Camera Model 

The fencing robot observes its environment through a 
single camera mounted on the third joint of the arm, as 
shown in Figure [T] We assume that the camera is capable 
of obtaining the 2-D pixel coordinates of the tip and base of 
the attacker’s sword. Thus, in this paper we do not consider 
the segmentation problem, but it is worth noting that such 
problem can be solved, for example, by using appropriate 
color markers. 

Given a point p G represented in a reference frame 
{x,y,z), we will denote each coordinate of p by [p]^, [p]~, 
and [p]-, respectively. 

Let pcam S SE{3) denote the transformation from the 
camera coordinate frame, {xcam, Vcmm -^cam), to the third-joint 
coordinate frame. The camera coordinate frame is assumed 
to be located at the center of the camera focal plane. Hence, 
the camera’s transformation relative to the world frame is: 


Gcami^d^ — 9d,li^d,l) 9d,2i^d,2) 9d,3{^d,3') 9cam- ( 7 ) 


The camera coordinate frame is only 2-D, hence the obser¬ 
vation p € of a point p S is modeled as: 


[p],_p = i^noG-i(0,)p, (8) 

where K G is the calibration matrix of the camera, 

which depends on the its intrinsic parameters, and: 


no = 


' 1 0 0 0 ' 
0 10 0 
Lo 0 1 oj 


e 


(9) 


is the standard projection matrix, which projects 3-D points 
into the 2-D camera coordinates. More details about this 
camera model can be found in Chapter 3 of [9]. 


C. Closed-Loop Architecture 

As explained in Section [I] we implement a closed-loop 
system consisting of an observer, which estimates the at¬ 
tacker’s intentions using a Receding Horizon Estimation 
algorithm, and a controller, which plans the path for the 
robotic arm to defend from the attacker’s actions using 
Receding Horizon Control. Both, controller and observer, 
work in a tight coordination. Even though the controller’s 
main objective is to defend the set S'taiget, its objective also 
includes a term to force the camera to maintain the sword 
within its field of view. Similarly, even though we could 
have designed an observer that simply estimates the future 



Fig. 2. Block diagram of the information flow in the closed-loop system. 


positions of the attacker’s sword, instead we designed an 
observer that describes the attacker’s movements using a 
fully actuated arm model, forcing our predictions to satisfy 
the natural constraints of an arm. 


Figure]^ shows a block diagram of the closed-loop system. 
We explain the details of the observer design in Section m 


and the details of the controller design in Section IV 


HI. Observer Design 

Once the 2-D camera coordinates of the tip and base of 
the attacker’s sword, pt and pt,, have been obtained via image 
segmentation, we are faced with the task of extracting useful 
(or actionable) information from this stream of coordinates. 
To achieve this goal, our approach is based on the following 
principles: first, predictions of the future positions of the 
sword must account for the non-holonomic constraints of 
the arm holding it; second, due to the fast nature of the 
movements in fencing, it is more useful to predict intentions 
than exact positions. 

We designed an observer based on a Receding Horizon 
Estimation (RHE) algorithm, which given a time horizon 
Tabs > 0 and a time series on the interval [t — Tobs, t] of 2-D 
camera coordinates previously collected, estimates the initial 
angles and the angular velocities of a kinematic arm model 
we arbitrarily assign to the attacker’s arm. Even though our 
opponent is assumed to be human, and thus our kinematic 
model might not match the actual parameters of the attacker’s 
arm, we believe the data produced by it is still better than 
using a model-less approach to the observation process. In 
particular, we chose to use the same 6-joint model we use for 
the robotic arm to describe the movements of the attacker, 
noting that if this model is in practice too inaccurate then it 
is easy to replace it with a different one. 

Let 0tj = \9a,iT ■ ■ be the joint angles of the 

attacker’s arm, satisfying the following ODE: 0a(f) = Ua{t), 
where Ua{t) G K® is the vector of angular velocities of the 
joints. We assume that Ga(Oa) G SE(3), the transformation 
taking points from attacker’s sword coordinates to world 
coordinates, is known. Note that by assuming that Ga is 
known, we are implicitly assuming that we know the exact 
location of the attacker’s body. We currently assume that the 
attacker’s body is static. Also, let pt(t) and Pb(t) be the time 
series of observations in 2-D camera coordinates of the tip 
of the attacker’s sword tip and base, respectively. Hence, at 













time t we solve the following optimization problem: 

min f A(t) ^ \\pj{t) - qj{T)\\l + ai\\ua{T)\\ldT 

Jt-n. 

S.t. 0a ('^) € [ 0max5 0max]; ^ [ '^maxi ^max] 5 

0a('7') = Ua{r), 9a{t “ ^obs) = <?, 

QjiT) = Ge”m(0d(T)) Ga(0a(T))pj, 

h ("t)] Q] (t) = KUo Qj (r), 

fe(T)]^ > £min, Vj e {t, b}, \/t G[t- Tobs, f], 

( 10 ) 

where A(r) is a forgetfulness factor putting more weight on 
recent observations, defined by A(t) = exp(a 2 (r — f)), pt 
and pb are defined as in equation ai, a 2 > 0 are constant 
parameters, and Emin is the closest distance an object can be 
from the focal plane of the camera. 

We solve the optimal estimation problem in equation ( [TOl l, 
at regular intervals of Ti„t > 0 seconds, by transforming 
this problem into a nonlinear programming problem via 
time discretization as described in Chapter 4 of [23]. As 
part of the discretization process, we assume that Ua{t) is 
constant over the interval [t — Tobs,f]. The intuition behind 
this simplification comes from the fencing rules [8], which 
require an attack to be a continuous motion towards the 
target. This assumption is also consistent with the speed of 
fencing movements, which are completed in the order of 
hundreds of milliseconds; in an interval of that length we 
can only acquire and process a few tens of images, hence it 
is better to use a simple movement model than running the 
risk of over-fitting the data. 

IV. Controller Design 

In fencing, an attack can be interrupted by contact between 
the two fencers’ swords [8]. The most basic form of blocking 
consists of the defender placing his sword between the 
attacker’s sword and the target area. Since the attacker cannot 
interrupt his motion towards the target area without ending 
his attack, this is often sufficient to achieve a successful 
block. We designed a Receding Horizon Control (RHC) al¬ 
gorithm stop the attacker’s sword with the defender’s before 
the attacker reaches any of the points in 5'target- The RHC 
algorithm works by iteratively solving an optimal control 
problem, with time horizon T^on > 0, every Tint > 0 seconds, 
and it achieves the desired objective after a careful design of 
the objective and constraint functions, as explained below. 

We define the blocking plane as the plane parallel to the 
defender’s sword with normal vector Xd- Thus, given a set 
joint angles of the defender 0^ G M®, a point p G and the 
function hbp(p, 0d) = (0d) {p-Gd{0d)Pb), the blocking 

plane can be described as the set {p G I hbp(p,0d) = O}. 
Our controller must maintain the attacker’s sword and the set 
S'target divided by the blocking plane to ensure the safety of 
the defender. Mathematically we write the condition above 
as the following constraints: 

hbp(Ga(Oa)pj,Od) > 0, Vj € and, 

^bp(^;0(i) ^ 0; ^ *5*target- 



Fig. 3. Diagram of the blocking plane (in (£di yd> ^d) coordinates), the 
projection of the attacker’s sword into the blocking plane (with length (pi, 
defined in equation ( |13) ), and the distance of the attacker’s sword ends to 
the blocking plane (their sum is (p 2 , defined in equation 0)- 


Since the length of the sword is finite, it is not enough 
to enforce that the attacker’s sword is separated from ^target- 
We must also be sure that if the attacker’s sword is close 
to the blocking plane then the defender’s sword must be 
within reach of the attacker’s sword. We achieve this goal 
by imposing minimum and maximum distances between the 
defender’s sword and the projection of the attacker’s sword 
onto the blocking plane. Recall that the blocking plane is 
normal to Xd- Thus, given p G and 0;^, 0^ G K®, if we 
define: 


^box(P; 0tt; 0a) 


f[G^^(ed)GMp-p]A 

[[G^^(ed)Ga(0a)p-p],J^ 


( 12 ) 


then the constraint described above can be written as 
bmin < hbox(Pj,0d,0a) < (>max for each j G where 

bmm,bmax G are the minimum and maximum distances 
allowed, respectively. The blocking plane and box constraints 
are shown in Figure 

To ensure that the defender’s sword stops the attacker’s 
when it approaches the blocking plane, we penalize the 
swords being parallel. Mathematically, we achieve this goal 
by including the following term in the objective function of 
our RHC optimization problem: 


V5l(0d,0a) = \Zdi®d) {Ga{0a)Pt “ Ga{0a)Pb)\, (13) 


which is minimized when both swords are orthogonal. Note 
that even though (pi is a non-smooth function, it can be 
easily implemented as part of a nonlinear programming 
problem using the traditional epigraph transformation (e.g., 
see Chapter 4.1.3 in [24]). 

We also penalize the possibility that the swords could 
avoid one another if the projection of the attacker’s sword 
onto the blocking plane is very small, i.e., if it is approaching 
in a “stabbing” motion. Consider the following function 
calculating the sum of the distances between the ends of 
the attacker’s sword and the blocking plane: 

T2(0d,0a)= X! {Ga{Qa)Pj - Gd(^0d)Pb)\- 

(14) 

Minimizing this function reduces the distances between both 
ends of the opponent’s sword and the blocking plane. Note 
that when ip 2 is minimized, the attacker’s sword becomes 











parallel to the blocking plane, and at the same time the pro¬ 
jection of the sword onto the blocking plane is maximized, 
thus removing the possibility of a “stabbing” motion. This 
function also helps ensure that the swords will eventually 
come into contact, since aim to reduce the overall distance 
between both swords. Also note that thanks to the constraint 
in equation ( [TT| l we can simply remove the absolute values 
from the formula of <^ 2 - 

Note that all the functions above involve the RHC algo¬ 
rithm using information provided by the observer to estimate 
the angles of the attacker’s joints, ©„. However, the quality 
of the estimations provided by the observer are directly 
related to the data captured by the camera, which in turn is 
related to the angles of the defender’s joints, 0^. Hence, we 
add a term to the objective function of the RHC optimization 
problem that is minimized when the attacker’s sword is 
centered in the 2-D camera coordinate frame; 


©a) 


iTnoG-^(0d)GJ0Jpb 

[G-ai.(0d)G,(0a)pb], 


Although it would be ideal to implement a hard constraint 
requiring the controller to keep the opponent’s sword in the 
camera’s view at all times, in practice this leads to an over¬ 
constrained problem which often has no feasible solutions. 
Therefore, we implement a soft constraint adding tpa to the 
running cost. 

Recall that at time t we know the defender’s joint angles 
an estimation of the attacker’s joint angles 
and an estimation of the attacker’s joint angular velocities 
Ua{t), the first from angle encoders mounted on the robotic 
arm, and the last two as results of the observer presented 
in Section [III] For prediction purposes, we assume that the 
attacker will apply constant angular velocities to all its joints 
on the interval [f, t + Tcon] equal to Ua{f). 

Combining the functions defined above, the results from 
the observer, together with dynamics constraints and a regu¬ 
larization term in the objective, yields the following optimal 
control problem: 


min J (73‘F3(0d('r),0a('r))+74 ||Md(T)|| 2 ) dr-l- 

+ 7l +l(0d(t/), 0a(f/)) + 72 
S.t. 0 (;(t) G [ 0 maxj 0 max]; '^dij') ^ [ ^max; timax] 5 

0d('r) = Ud{T), 0a('r) = Ua{t), 

/lbp(Ga(0a(r)) Pj, 0d(T)) > 0, 

/lbp(s, 0d(T)) < 0, 

^min ^ hbox{pj,Qd{T),QaiT)) 

— ^max; 

Vs e ^target, Vj G {t,b}, Vt G 

(16) 

where tf = f+T^on, and 7i, 72 , 73,74 > 0 are scaling param¬ 
eters. Similar to the optimization problem in equation ( [TOl i, 
we solve this optimization problem every Tint seconds by 
discretizing it using the techniques described in Chapter 4 
in [23]. 


TABLE I 

List of the parameters used in the simulations. 


Parameter [unit] 

Robot segment lengths [mm] 
Sword length [mm] 
©max [°] 

©0 n 

Umax [°/s] 

Camera focal length fi [mm] 
Camera pixel size cTp [/rm] 
Camera resolution [pixels] 
Ton [ms] 

Tbs [ms] 

Tint [ms] 

Tisc [ms] 
ai 
«2 

71 

72 

73 

74 
^min 
hmax 


Value 

[50 250 300 75 so] 

880 

[I8O 150 190 190 220 360[^ 
[O 135 95 0 115 -15]^ 
[350 350 400 450 450 720[^ 

12.5 

4.5 

1600 X 1200 
100 
50 
15 
5 

10-2 

10-2 

10-2 

1 

10-4 

10-4 

[-500 10]^ 

[500 600 [^^ 


V. Simulations 

We simulated the behavior of our closed-loop scheme for 
a collection of predetermined movements of the attacker. 
We modeled both defender and attacker arms based on a 
FANUC robotic arm model LRMate 200iC [25]. Also, we 
modeled the camera mounted on the defender’s arm on 
a Point Grey camera model Flea 3 [26]. We discretized 
both RHE and RHC optimization problems, described in 
equations ( fTO) ! and ( [T 6 | ), with a step-size T^isc, and we solved 
the resulting discretized nonlinear programming optimization 
problem using the SNOPT library [27]. The simulations 
were programmed using the language Python, and they were 
executed in a two-processor Xeon E5-2680 computer running 
at 2.7 [GHz]. A detailed list of all the parameters used in 
our simulations is shown in Table U Note that the camera 
calibration matrix is K = diag(/£ Up,Up, 1), where fi is 
the camera focal length, and tjp is the camera pixel size. Also 
note that even though the sword length is 880 [mm], the upper 
limit for the box constraint in equation ( [T^ along the z^^-axis 
is 600, since this value reflects the fact that attacks are hard 
to stop in practice with the tip of the defender’s sword. 

We include simulations under two different scenarios 
below. In the first case the attacker’s joint angular velocities, 
Ua{t), are constant over the whole , the second where Ua{t) 
changes over time. We also present results validating the use 
of the penalty function (^ 3 , designed to keep the attacker’s 
sword within the held of view of the defender’s camera, in 
the RHC optimization problem in equation ( [T6| ). 

A. Example 1: Ua is constant 

We begin with a sample trajectory where the assumption 
that Ua{f) is constant holds. The initial joint positions for the 
robot and the opponent are given by 0 ^( 0 ) = 0 a ( 0 ) = 0 o, 
with 00 given in Table [I] and the opponent’s trajectory is 
defined by: 


Ua{f) = [-21 -210 -400 345 -63 o]^. 


(17) 










73 = 0 


73 > 0 






Fig. 6 . Attacker’s sword (solid red), observer’s estimation of its position 
(solid blue), both in z^) coordinates, and their projections on the 

blocking plane (dashed red and blue, respectively), for two instants of time. 
The top row shows a simulation where 73 = 0, and the bottom row shows 
a simulation under the same conditions, but now with 73 = 10 “'*. 


This trajectory defines an attack to the robot’s left side. 

Figure shows the optimal trajectory computed by our 
closed-loop algorithms for this scenario. The robot success¬ 
fully blocks the opponent’s attack by achieving blade contact 
at time tf = 145 [ms]. 


B. Example 2: Ua is not constant 

This example shows the behavior of our algorithm when 
the assumption that Ua{t) is constant does not hold. The 
initial condition is identical to the previous example, and the 
opponent’s trajectory is defined by: 


Ua{t) 


[-68 -200 -400 -355 280 o]^, 
[34 -309 -382 244 -329 o]^, 


if f < 85 [ms], 
if f > 85 [ms]. 

(18) 


Figure [^ shows the optimal trajectory when we do explicitly 
control for observation. In this case, the robot successfully 
forces blade contact at time f/ = 160 [ms]. 


C. Effect of Active Perception in the Control Loop 

As described in the introduction, our motivation to im¬ 
plement this particular control loop was to show that it is 
possible to extract all the necessary 3-D information from 
a single camera if the data is analyzed in the context of a 
dynamical model, and if we can improve the capture of data 
at the same time we defend from the attacker. 

The first objective, extraction of information using a 
dynamical model, is achieved by our observer (explained 
in Section IIIi, while the second objective, dynamic im¬ 
provement of the data capture by the camera, is achieved 
by adding the function defined in equation ( fTSj l, to the 
objective function of our optimal control problem, defined 
in equation ( [T6] l. 

We evaluated the relevance of ip^ in the closed-loop 
performance by simulating the configuration described in 
Section jV-Aj with 73 = 0 and with 73 > 0 (using the 
value in Table [I]). Figure [^ shows the attacker’s sword next 



Fig. 7. Distribution of the estimation error when 73 = 0 vs. 73 > 0 for two 
measurements: attacker’s joint angles (left), and position of the attacker’s 
sword tip and base (right). Note that the attacker’s angle 6q is omitted since 
it rotates the sword, hence it is irrelevant for control puiposes. 


to our estimation for both cases, 73 = 0 and 73 > 0. For the 
configuration described in Section V-B the camera quickly 
looses track of the attacker’s sword, so the optimization 
problems did not produce meaningful results for 73 = 0 . 

We also quantified the effect of explicitly controlling for 
observation by looking at the error the observer makes 
when 73 = 0 and 73 > 0. Figure [^ shows the estimation 
error empirical distribution in each of the attacker’s joint 
angles, and in the positions of tip and base of the attacker’s 
sword. The distribution was estimated using 20 different 
simulations, each with a different attacker input Ua chosen 
such that the attacker’s sword successfully reaches a point 
in S'taiget provided the defender does not move. Although the 
worst case errors in Figure [7] are comparable, particularly in 
the location of the tip of the sword, the median error is much 
lower when 73 > 0. In fact, the worst case arises when the 
camera loses sight of the attacker’s sword, but this situation 
occurs significantly less when 73 > 0 . 


VI. Conclusion 

We have presented a method for simultaneous sensing 
and control of a robotic fencing arm with a single camera 
attached to it. We extracted the useful information from the 
images captured by the camera using a Receding Horizon 
Estimation algorithm, and we controlled the robotic arm to 
defend against an attacker using a Receding Horizon Control 
algorithm. We included in our Receding Horizon Control 
algorithm a term to maintain the attacker’s sword within the 
field of view or our camera, thus actively improving the data 
captured, and by consequence, the quality of our estimations. 

Our focus in the near future includes the real-time im¬ 
plementation of our closed-loop system using a real robotic 
arm, and the use of an heterogeneous collection of sensor, 
rather than a single camera. 
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